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The design^ development, and evaluation of a 
computer-assisted instruction (CAI) program in first-year college 
chemistry is described in this report. The program was centered 
around the concept of chemical equilibrium and used a systems 
approach with explicitly stated guidelines* The purpose of this 
experiment was not to measure the effectiveness of the course 
content^ but rather to investigate the usefulness and effectiveness 
of the particular developmental processes and design techniques that 
were employed. The results of the program revealed basically that 
students with low mathematical capabilities upon entering the course 
take longer to finish the prescribed sequencer and that a student's 
criterion score is inversely related to his pith length through the 
course, (Mc) 
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1 N T R 0 D U, C T I 0 N 



Thy uso of computer-assisted instruction (CAI) as an innovativo 
modium for individualizod instruction and as a form of instrumGntation for 
oducational resGarch is widely recognized. There have been, howevar, few 
ufforts to develop materials that hava a course-related GontGnt and also 
have Gxplicit instructional guidalines ani.easily manipulated variables. 
This seems unfortunats, since findings obtained in the learning laboratory 
under experimental conditions and with contrived or unrealistic content 
may not be goneralizable to the actual classroom or to an individuaiizod 
instructional situation. In the dGvelopment of this progr.im an attempt 
lias been made to include both of these potontial contributions of computer- 
assisted instruction. The program itself is justified primarily in tormH 
of the instructionai neGds of specific students j at the same time, both 
tue content itself and various of the instructional features were influ- ' 
encGd by considerations of research utility. 



The study of scienco should devfulo]) the student's ability to ap])ly 
jntific gencraliaations to a variety of diffGront situations. Even for 
thotic students whose primary interest lies hi the liberal arts, such appli- 
cations in the physical sciences at the colluge level should be, m part at 
least, quantitative in order to reflect adequately this aspect of scienco. 
For the student who is majoring in one of the sciences or in engineering, 
these quantitative applications assume a much greater significancG in the 
introductory college courses because both the remainder of his preprofos- 
sional study and also his professional activities will require use of them. 

Within the field of ehemistry, there are numerous principles and 
and generalizations that form the bases for quantitative applications. One 
group of these concerni^ ahmiaat &quiUbHum, a topic which is generally 
included in first=year college chemistry and especially in those courses 
designed for students wiio will pursue further study in sciencG. At The 
University of Texas at Austin; dynmHa equitibriwn is one of the araas 
accorded major emphasis in freshman chemistry, and approximately five to 
six weeks of lecture time are usually allowed for it. 

Two factors in addition to its relGvancG for students of firsts- 
year college chemistry were considered in chbosing the content for this 
particular program. In the first place, the application of equilibrium 
principles appears to be typical of a variety of quantitative applications. 
Also, the terminal objectives are clear-cut and readily definable, and the 
task analysis is a relatively straightforward one. 
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If situdonts aco to acqu.i.r.. KkUl. iii in.ikiru; ..-u.int; i L^.tiv... 
c;..t.i.un:j of sciontific prinuii.los, it itj dusirdule d'tnom to [.ractjc..' 
SUCH applications for thomseivtis. vn thu othor hand, auidance ami addi- 
tional instruction Should bo avaiidblo to the stuciont wiion they aiuj nou-ded 
In the past such supervised practice has frequGntly bcun provided thfou.jh 
small-group discussion sessions soparato from iho rcyular' lecturos . v„ny 
institutions, including The Univorsity of Texas at Austin, hdvo now uiis- 
continuod thia practice due to the scheduling dif ficuUios cncountGr.Kl 
with largfj numbors of studGnts and to the shortage of suitable instruc- 
tional spaco and of qualified personnel. Instruction in quantitative 
applications and in tho handling of numerical problGms has, tliersforo, 
boen incorporated into the lecture sections, each of which ordinnrilv' 
enrolls two hundred or more students. Thus, too much of the responsi- 
bility for actual practice in these skills must be placed upon the stu- 
dents thomselves , 

First-year students in collecje chemistry, fivun those wno are 
planniny to major in one of the sciences or in ongineering, typiCdlly 
present a rather wide variety of backgrounds in science and matnematjrq 
aa well as a range of individual aptitudes. Theso difforencea ijcconie 
parLLCulariy apparent in nuinerical applications wiiere not only chemical 
concepts but also iriathnrna tical skills must bo called upon. This sunqests 
the need for ■Lu.ii^JUluali:^atiu}L , where tius term is used somewnat broadly 
to dQScribe any tyj)0 of curricular adjustmiant that is based upon learner 
cnaracteristica. 



Conipu ter-A osia ted Ino tirua Hon 

One possible way of meeting the student's need for individual- 
ized instruction is through the use of CAI . Such instruction permits the 
student to advance at his own pace until a criterion level of performance 
is reached and to do so, to a largo extent, at the time of his choice. 
It also offers a number of other advantages for the student, advantages 
that are generally unavailable with alternative modes of instruction! in 
the first place, the student proceeds througi-. the program along a t^athway 
that IS directly ralated to his own responses. This taranching can" be made 
very complex and extensive with computer-assisted instruction because the 
computer can be prograiimed to make decisions based not only upon a particu^ 
lar response to a specific item but also upon cumulative responses or 
response patterns to a series of questions or problems, m addition, the 
assiatance given to the student can often be made quite exulicit, since the 
autnor who is also experienced as an instructor can anticipate many of th<=' 
errors the student is likely to make and can provide guidance that is di- 
rectly related to a particular incorrect response. This avoids, for the 
student,- the frustration of knowing that an answsr is wrong but having no 
real clue about the source of error. A further instructional advantage 
results because the student can be encouraged, even forced, to proceed on 
his own as far as he is able; thus, he is more likely to develop real atailitv 



ERIC 



2 



to auAy prir.cir.ios. rat.-.or than ;-or..iy r,kin in ru.-aJUn,.- .. .. ri,.u ,.r ..v 

.no same f.oxi.Uity tnat i.rouuc. .uUxpIu ..tinvay^ ti.r.u.a tn. Vro..:,;;"":,; 
m..n tound La niynly motivatiny for nuny student.. Th.y fooi • w. 

in.tructxanai material., rather tiKu: boi,.; i,.,.r«oM.l .,nd moen.n i-^a i ; .p-. 
r.u;,«naim, to tnoir individual need, and dxr f icultxo. . A fin.l, ,otc'.. i 1 
important, advantagu is .ssociatod wruh ci.u u.e of tn. com.utor in .cm."" 
. xnn.ruction. Bocauae these devices havo widospruad uuplicaMcn. ,n' tn^^ Z : - 
onces ana in onyxneering, early OKuerionco with t:hom is valuablo to tnn .tu^ 
u..nt WHO IS planning to anter one of tneso fields and who wiil, th.-r.-raro " 
prouably be ca.J.od upon to make more direct use of them later in hin study. 

_ m addition to Its advantayes for the studont, a computer-^as.u tHd 

instructional program of this typo can also have potential usefulness .n ' 
educational research. If thi. potential is to be realized, tne do.i.r/r.c,= 
cesa usee to produce the instructional materials should »e clearly ".pec-l ■ led 
and tne program xtself should nave identifiable Instructional var iaul es' , bat" 
c^n bo manipulated at a later time. An attempt has been made to Inc-orp.;., . 
the.o^ features into tne computer-aB.i . ted program taat wa. ueveloped tor thi. 

Because of Uie instructional neods of students and because nf''ne 
poBsiole utility of comyuter^^aasis tod instruction in meeting tne.e ne.d^ ' " 
a program designed to help studente in first^year college ehomist;y acquire 
^kill in applying tne principles of chemical equil ioriuiT, to problems involv^ 
ing aqueous solutions has been developed and evaluated. An instructional" 
«ysterns approacn was ueed in daveloping the program, and explicitly stated 
•guidelinas ware followed in the preparation of the matariala them^elve^ ' 
Iho final program was written in Coursewriter III for use on a timci-.h^rinq 
basis with an IBM 360/50 computer system.. In addition to the developmental 
testing with individual learners and very small groups, the final computer: 
assisted instructional program was evaluated with a group of students from 
Cnemistry 302, the fir.t-ycar college chomistry course at The Univerni ty^,f 

uxas at Austin. Particular attention was paid to evaluating the instruel 
tional program m terms of its own objectives and expected outcomes, the 
latter being derived from the systems approach that was used in the develop^ 

The principal amphasis of the study was not on demonstratinq that 
computer-^assisted instruction can be effective in the content area that^aJ 
c),osen but rathor on investigating the usefulness and effectivenass of thr 
particular developmont processes and design tecnniques that were employed 
Thus, wnile tne results are directly applicable only to a single specific 
program, they also illustrate in a more general way" the feasibility of com^ 
bininy instructional guidelines and explicit decision rules with classroom^ 
oriented content, it is felt that this approach provides the^kind of ' 
empirical evidence that is a necessary part of in.structional research. 
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P R O G R A M 



D E V E L 0 P M K N T 



AND EVALUATION 



A systems approacn (tJraut, lyG7) was used m dovolo[)ing the matcr^ 
iais for tho instructional prograni. The implemontation of this approach hds 
beon dGscrj.bed by several writers (Briygc; et al,, 1967; GlasGr, 1965; E^under- 
soH/ 1971), all of wnom include essentially the same isrocedures. The dc^volop- 
ment process that was carried out as a part of the present study has, for 
clarity of presentation, been divided into the followinq steps: 

(1) Specif icatisn of tne terminal beiiavior and of tha entering 
skills. 

(2) Description of trte intermediate objectives and arrangGment 
of the various objectives into an appropriate learninq 
hierarchy. 

Development of an instructional sequence. 

(4) Design and preparation of the instructional materials. 

(5) Developmental evaluation and revision based upon the 
results obtained thereby. 

Evaluation of the completed program. 

This sGparation of the development process into several components does not, 
of course, imply that the activities werra carried out independently of each 
other. On the contrary^ there was considerable interaction among the various 
parts as development proceeded. 

Intarmediate design products developed in the process of preparing 
the instructional program were formulated and explicitly stated at appropriate 
points. These were concerned with sequencing, with individualizing mechanisms, 
and with the structure of the materials themselves. 

Tep^ninal B&havioi* arid Entgring Skills 

The first step in developing the instructional materials roquired 
the spacif ication of the terminal behavior and of the entering skills. The 
terTninal behavior, of course/ describes wna; the student should be able to 
do at the conclusion of the instructional prograin. It follows rather directly 
from the more inclusive goal of science education that students should be able 



to aj.ply Bciuntific ].ri.ncji i.Ios to ^lO^LiLc ^ituationti w/:on a Lujfinitic ,^ot 
of j.ri nc ij-aoja ana mn\u particular kinun o: Hituation^^ aro in trnduc*:-d , 
TiiUH, tnci tDrrninal IjCihdvior doycribtjs Htuclont ah i ii ti^'S to ai'i iy ihu CQn- 
c^i^ta anu principiua of cnoniic£il aquiiibriun; to problonib involvinsl aJUuou;:; 
solutions of sliglitiy solublci stronu aiectrolytos or aqueuus solutiojis of 
soIuLle woak eiectroiytos , TaDle 1 contains' the s^t of relotud ab^Li'C ti v.;^; 
anci Lho mininium porformance critorion tnat, taken togothur, ciescritaG thu 
tanninal behavior. 

The ontoring skills describo certain abilitit^s that the target 
population ia expectod to posaeas. They also serve to sot a lower limit 
to the learning iiierarchy, for it should not be necessary to instruct the 
student in those skills tiiat lie i. prQsumud already to posSu?;is. The yon- 
eral target population for this program consists of first-year colleyc 
chemistry students who are planning to major in one of the sciences or in 
enginearing; more specifically, it is made up of students who are enrolled 
in Chemistry 302 at The University of Texas at Austin, It was ai^siuned that: 
the typ-ical n^ernbor of this population would have had, as matliematical ijrepa = 
ration, at least two years of nigh school algebra and that he would also have 
either successfully complotod one semQstGr of collugo chemistry or demon- 
strated comparable achievement through advanced i?lacement . It is readiiy 
apparont that such students are likely to possess a rather large number and 
variety of skills. Fortunately, it is not necessary to include all of these 
an entering skills but only^ those that form an explicit part of the iearn= 
xng nierarchy. it now becomes obvious that the learning hiararcny and the 
list of enter ing skills are closely related to each other. The two were 
actually developad simultaneously, and they have been separated here merely 
for convenience in discussing them. The entering skills that were identified 
as relevant to the other objectives of the instructional program are listed 
in Table 2. Thesu entaring skills, like the terminal benavior, have been 
described by moans of behavioral objectives. It is seen that only two of 
tna ontering skills are related to cnemistry, the otner three being mathe- 
matical in nature. 



Intarnwdiate Objeativss and Lmpning IliBrapany 

In the second phase of the development process those intermediate 
objectives that need to be attained by the learner in order for him to 
progress from the entering skills to the terminal beiiavior were identified 
and defined in operational terms. The identification was accomplished by 
asking, in the manner first suggested by Gagnl (1962a) i 

What learnings, concepts, and skills will the 
student need in order to be able to perform 
the calculations required by this part of the 
terminal behavior? 

Th£ instructional designer, in attempting to answer this question, utilized 
both logical analysis and prior experience in teaching these specific appli- 
cations. Table 3 has statements of the intermediate objectives. 



Tab]*' 1 



■ i i j.Vi.;! V ijiO i!'jU*jOUH hiOiULionK tit Kulublrj wiKik t? lt?ct; r n 1 y i;* .^h . Tlv:- Htudont will dui'^'.'^iiii 1 1 ai: 
iiumr'r irjn L hoIuILqiil^ Lo |.rubiensi oL tne fal lowing ty|.-?}S : 
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jt-'HE rat i Oil U') nf oipj^ or LjLjtih ot thu dinHolvoti ipiig 4JJ a ^Al.ui^ivd -iiiiui. ion of j 
Btronfj uioctrolyte in diuLiilod watpr, tho studont will by jblo. to .'^iiv-uLiti' Lhc 
soiuhtllty product of the ulectrciiytu , 

'I'^ibi Given t.hy solubiiity pruduct of a yli^jhtly solublt; stionq o ioc tirnlytij , tJif studi-nt 
Wi i 1 ho iih 1 a to t'a 1 trU 1 a tii tho mo Icir cone t> n t rci t i gng f? r' Qi\c n ion 1 C r? o c i o y ; n o 
MaturntDa jiolut idn af tiiD o 1 uc Erply tt? in dintiUod waEur. 

T=*^. Givun ths;* i^piubiiity |?toducit of a eliqhtly Solubln gttong ulcctrolytG and tho 

n-Kji-nr uoncon t rufe i on ot onu of the ions a^rLVi'd fi^om thy u loy t ro I y ti> ^ tho ntudfir,t 
Will be tibiu tQ ^al^yulatu the cuncdntirat lion ot thu othi^r ion. 

T-'ia. yi¥t>n the dissociatiQn constant and thu nominal ccncontrat ion of a wuak <ii-ld iot Uiiit.) 

in dLgtiiied water, the student will be able to caie'ilatf-' any Or all uf tjio fcillowiiuj: 

(i) Tliii concentrationg of all ionic ^pqqLqb. 
i?,) Tho p!f and/or the [■Oil of the ^^olutiyn. 

Given thQ pH and the nominal concentration ot a weak acid (or baso) in digtiilud 
water, the student will bu able to leu late the disgaciation constant for the 
wnak aeid iov base) , 

T='1n. Given the molar eoneentrationsi nf acid (or base) and of ialt m a buffer golutioii and 
thcj dissociation eenstant sf the weak aaid (or bagg) , the student will be able ta 
calculate the pH of the buffer. 

T=4b, Given the pH of a buffcir flnd the disncciation cOnitant of the weak acid (oe- base) f 
the student will be able tg calculate the relative eenceritrations of acid (or ba^e) 
and salt. 



T^4e, Given tiir- pH of a buffer # the ionizatio 
eitriei' t!ie CQncentiation of tiie acid or 
will be ably to calculate tht fefnaining 



i constant of the weak acid (ar base) , and 
the concentration of the salt? the Btudent 
concentration (of acid/base or gal t) ^ 



The student's ability will be ei/aluatod by meani of a po^ttest at the conolusion of the instructional pro= 
qram* He should hm able to obtain correct solutions for 70'^ of the probleftig rcpresentinq a Jiampliny of 
tiie abuve types. 
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l'=L TlH. sLuclanl vili b--. .ih]c. to wrue .ind if;Lur| r..i a jKu:ni.,nL uau,u inn .s.^^.^r 5 m » 
.;i%v'n soluLiDn Df inhi-jLiQi* prpcqys. TiU.' utudont will cWmonst? .uo hi^ ail lily 
Jri uru? gr bath of Ljid fylldwiiii;^ Wtiyys 

(n GivMn tliu Coriiiula tor a sliqhEly t^vjluhlo or Huluhlf- ntronq oioctinlyu^ 
or fjr a f^i^luhLu wo^ik cietJt.fulytP , thu atudunt will writr^ a LaLuuM^i 
eiigmiu'^'il uquation to rupresenE t]\ti nulutLon ar lorii-^iitipn prDco^i;. 

(2) Fur ri givqn amount (or conconCrtition) of di.ggolvijd gr igni?.L'd oiiu-t lo IvL*- , 
Lh'.i studtint will nr abla to state tliu iiinount {gr t;Dnct^ntraEi.jn) of uciuh 
ionic apeeieH tornied cmmA/ut the tetdl oinount (or aoncontrn t iojj ) cf 
L'n sglution^ 

I'hc HtMiii'UV wi n Jm- ablo tn dcfinn njul UBu thf LcrTn "moidr" Lw Jubi/Lii-u' thu 
ftoncpntratioii a agiuEiun. Thi> □tudtint will demongtrato ability in ono 
gr moro of thu fpligwinej wMyist 

(1) Tho studnriE will Ntatg in writiny will idgntify fioni arnony a group of 
written staEcmunte the dGfinitign of "molar," 

(A) Givun the? namj or formula fer £j uubjitancc and iuitflble informiltion fnr 
4£Jtt-»rmininy ita formula mass, tlie itudunt Will dDsctibt' thfa contontKi yf 
or a suitably mot had far prepiariny ti pglution of the givnn substancu 
haviriq tiuy yiven molarity, 

Tho gtudnnts , with the help of a table of eomman logari chins , will ho able to trans^ 
fonr. auy nv.^r. nuiTibor, to log U dnd any yivan Isgarit^im, logN, to 



P=4i Thsj student will be able to generate and manipulate numbori exprossed in o^panentitil 
notation, Ht? will demdngferate hif ability in any or all of the fallowing Wiiygi 

(1) Civen a deglnuil numtjorj the Student will write the equivalent fgrm of it in 
standard ieientifle notation. 

(2) Given a number in e^penential fonn, the studgnt will write tho equivalent 
decimal nujnbor, 

{3} Given tin at it lime tie problem involving thg opoirationi of addition, gubtraqtign^ 
mul t ipl iGation , division, raising to a power, and/or extracting a root in which 
ssRie or all of the nUlnberi are expreseed in Q-<ponential notation, the student 
will solve the problem and write the solution in exponential form, 

P^5^ The student will be able to golve first= and ieGOnd^degree equationi of the 
following types: 

an + Ly 4 c ^ 0 and qk" + by^ + gk + dy f e ^ 0 

Hq will demgnetrate liis ability in ono or both of tho following wayi? 

(1) Given an equation of the above? type, the gtudent will solve it for either 
variable, 

{2} Given an equation of the above type and infermahian from whieh nuinerical 
Values for the eonstants and for one of the variables may be deduced, ths 
student will calculate the numerical value of the other variable. 



The gtudgnt should be sble to answer carreetly '.0% of a group of queitigni and problems repreienting a 
gampling Of the above objectives. 



= -1. Tho studont will DC ably to define tho Qquilibrium constant mathoniat icall^ 
ffo will demonstrate his ability by writing or identifyincj tiio oauilibrium' 
OKprGssion for the gGnoral gasGous reaction: 

aA + bB cC + dD 

;-2. TiiG student will have a functional undors tanamg of the concoots and 
principles associatGd with cnemical equilibriuin and the equilibrium 
constant. He will dGmonstrate his ability in one or more of thm 
following ways: 

(1) Givon the equation for a specific equilibrium reaction, thi^ studunt 
will identify those substancGs whaso concentrations appear in tho 
Gquilibrium constant. 

(2) Given the equation for a spoeific oquilibriimi raaction, thG 
student will write the expression for tho equilibrium constant, 

i"3. Givun the equation for a specific cquilibriujii reaction, tno student 
will describe qualitatively the eifect that a change in conditions, 
such as concQntration, tempera turc, and so on, will have on the posi- 
tion of the Gquilibriuin. He will also identify or state in writing 
the principle that is being used together with its name. 

^-4. The student will state in writing or identify the definition of pH." 

-5. The student can use the concept of pH and will demonstrate his ability 
in one or both of the following ways: 

(1) Given the hydrogen-ion concentration of a solution, the student, 
with the help of a table of common logarithms, will calculate 
the pH of the solution. 

(2) Given the pH of a solution, the student, with the help of a table 
of common logarithms, will calculate the hyurogen-ion concentration 
of the solution. 

.i>6. The student will be able to write the expression for the ion-product for 
water, to specify the numerical value of this constant at room tempera- 
ture, and to calculate the hydrogen-ion concentration, the hydroxyl'-ion 
concentration, the pH, and/or the pOH wnen given any one of the preceding 
quantities. 
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Tno ontoriuq skills, Li„. i.nt^rnu..d Kjt.. .^n Uv,;ii, uivi t:.. 
'...j-cLiv-. included in tno terminal »wh.iv:<.r wuru u.^n i,,in,.u tou.-ti.T mi., 
u uarnin.j iiicirarchy. rho ludrnin^i hierarchy snow;.; d , .-Jjrain.it; i , •„[ 1 - t 
rul.itiynsiiii.s britwoen tho various oujc-cUv...^ und indicaL-, w.iich o,: ....i ;v...= 
...re consioft^rud to he proroquisito for th.. m.ii.Lcry of otl^ors. i,.r.. uM.ijn 
botn ioyic3i analysis and ^nipiricdl observations dorived from trachm-r' 
ox,H.riQncc. wore relied upon. Figure i presents the loarniny hicrarrhy- 

no lotterii and numbers that are used refer to the various c^hjoctivni .v, 
listed in Tables 1, 2, and 3. . 



J. tit] 1 c 

lol- 



ImtpuaHomt SaquQnae a}id IndtvUnaliiUn^^ •iecnanimn 

The next stt?p in dovoloping the instructional nvitprials rr.anriH = 
the translation of the learning hierarchy into an ins tructiona] Returner. 
bmcQ tho present state of injstructional desion theory doGs not offer .inv 
overall rationale for this operation, common .sense , intuitinn, end th- Jo 
of the learning hierarchy itself provided moat of the directions- Th-^ f 
lowing yanaralizations guided the planning of the insitructional scciuenee 

(1) Those terminal, objoctivoa, or coiiiponent jiartri of tho tnrmin,. 1 
behavior, that are most closc.y related to each otlier u.iould 
be placod in nearest proximity in the soquence. 

(2) Related terminal objectives should bo presentBd in what 
appears to the designer to be the order of their increasing 
complexity. 

(3) The various intermediate objectives should be introduced 
at the point where they will ho most immediately useful 
to the learner. 

(4) Materials not related to either the intormodiate or the terminal 
objectives should be placed in the instructional sequence at 
those points whara the student first has need for them £5uch 
materials would include, for example, information about the 
program itself and about the conventional forms to be used 

in constructed responses as well as short reviews of snlocted 
entering skills. 

(5) Opportunities should be provided for the student to practice 
tne terminal objectives not only singly but also in related 
groups (Gagne, 1970) . 

Unless the instructional program is to be completely linear vdth 
all students using exactly the same materials in precisely the same order 
the instructional sequence itself should be a variable one. individualizlnq 

mechanisms make it possible for students to have different pathways through 

a set of instructional materials. These mechanisms may be entirely under" 
the control of the learner, i.e., the student may be allowed to choose his 
own pathway through the instructional program. They may, on the other hand 
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FIGURE 1. --Learning Hierarchy, 
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bt wholly under program control, in wiiich case tho program itself contains 
rules and directions for defining the individual studf-nt's movumunt through 
the instructional materials* It is also possible, as was done in the preso 
program, to comiuno the two forms of control, leaving some docisionH to tho 
studfjnt and reserving otherK to the pirograin itself. 

For those individualiziny mechanisms that wore retained under tht- 
control of the program, the following decision rules were used! 



(1) A tutorial sequence that is related to an intermediate 
objective is to be bypassed if the student can demonstrate 

'adequate mastery of the principles and concepts involved. 
The demonstration of "adequate mastery" is defined as re- 
sponding correctly to at least 85% of the criterion items 
presented prior to the tutorial sequence. 

(2) A practice sequence that is related to a terminal objective 
is to be bypassed if the student can demonstrate adequate 
attainment of the behavior involved. The demonstration of 
"adequate attainment*' is defined here as solving without 
error a problem of the same type in the corresponding 
tutorial sequence . 

(3) A problem series within a practice sequence is to be terminated 
when the student can demonstrate adequate attainment of the 
behavior involved. Demonstrating "adequate attainment" is, 

in this case^ defined as solving without error the precedinq 
problem in the practice sequence. 

(4) Where a practice sequence contains problems described by 
several related terminal objectives, the student should be 
presented only problems that are similar to the items he 
missed on the immediately preceding mastery test. 

These decision rules served the dual purpose of insuring that the student 
mastered one objective before moving to another one and of relieving him 
of additional instruction in an objective that he had already mastered. 
This emphasis upon mastery is in accordance with the theoretical position 
of Gagne (1962b) and with the empirical evidence accumulated by Silberman 
and Carter (1965) . 



For a computer-assisted instructional program, the learning 
sequence is best summarized by a flowchart showing the orderly progression 
through the major components of the program and identifying the principal 
points at which the decisions governing this progression are made. Figure 2 
IS a highly simplified and abbreviated form of such a flowchart for the 
present progr™. The central portion of this figure contains those parts 
of the instructional sequence that are used by all students, including both 
the tutorial materials related to the terminal objectives and also the 
various diagnostic and mastery tests. The tutorial segments that provide 
instruction in the intemediate objectives are located on the left side. 
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and the groups of practice problom^^ far t;h<-' t^^rmlnil -bjoctivos :irn .^r: tnu 
riant sido. The total program contains two didgnostic toscs and thri^o 
tutorial seqUGnccs for the interniodiate objoctlvoy, two mastery testn 
ovor groups of rGlated tentiinal obj^^ct ivus^iid two vrautica seuucnculJ 
as&Qciatod with these, and a final critorion test over tnu entiro terminui 
behavior. Thero aro also various auxiliary tutorial sc^quoncos rulatud to 
the terminal objectives. These are accessed from tlie main tutorial hc= 
quencGS Gither at the option of the student or by the Drogram itself: whoro 
the student's responses indicate the need for supplementarv instruction. 
Finally, the instructional program contains a few brief review sequences " 
for some of the mathematical entering skills. The student decides for him- 
self whether or not to make use of these. 

The prograjn is constructed so that the student is not permitted 
to go through any part of the instructional materials more than one time. 
Where normal operation of the decision rules would lead to the repetition 
of a sequence, the student is told to see the proctor for individual helo. 



InHtmotional MateiHals 

The actual writing of the instructional materials can be con- 
sidered as the fourth step in the develo}3ment of the program. This v%?ari ^ 
actually begun as soon as the general outline for the learning sequence . 
had been decided upon, and the writing itself suggested changes in and 
provided detailed structure for the learning tJequence . In addition, the 
information gained through the developmental evaluation of the early 
instructional segments guided not only the revision of these segments 
^ but. ..also the initial construction of the later ones. There is, thus, 
much interaction betweGn the proce^jses of arranc ' ng the learning sequence, 
preparing the instructional matarials, and observing student use of the 
various parts of the program. 

Several generalizations, some of v^hich were suggested by Gagne's 
Urn CondiUons of L&miing {1967} , served as guidelines for writing the 
instructional materials i 



(1) When a principle is to be learned, instruction should provide 
for recall of the relevant concepts. 

(2) The student should be encouraged through verbal cues to 
organize and grasp the meaning of the principle for himself. 

(3) When a principle is to be applied, instruction should provide 
for the recall of the principle, if such recall is necessary. 

(4) The student should be encouraged to apply the principle for 
himself, and directions should indicate procedures without 
presenting solutions, 

(5) Varbal cues, where possible, should be directed toward a 
specific learner difficulty, 

(6) The instruction should incorporate both large and small steps. 
As long as the student is ablQ to demonstrate progress, tteps 
should be large; when the student encounters difficulty, steps 
should uecome smaller, and repetition should increase " 
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Since this program has been designed to porfonn a re?al 'tc-atliin j 
function with respect to the application of equilibrium principles, the 
instructional sequences pertaining to the torminal objoctivos utr. quito 
comprehensive. The student is Gxpected not only to obtain the correct' ^ 
answer tj the particular problem, that has bt-en presented to him, but also 
to analyze- the general aspects of the equilibrium reaction under consider- 
ation, and so on. On the other hand, because it was anticipated that the 
student would receive additional instruction about the concepts and princi- 
plGS of chemical equilibrium from other sources, the materials selectGd 
for the intetTTiediatc objectives were severely limited in scope and only 
those aspects of immediate usefulness to the student were included. Even 
this restricted instruction, of course, was omitted for the student who 
could show that he had already mastered the nacessary principles. 

The instructional materials consist of questionH to be answered 
and problems to be solved by the student, together with a minimum air.ount 
of expository text to be read by him. After the student has resDonded 
he receives some kind of comment on his answer from the instructional pro- 
gram. This might be nothing more than mere confirmation in the case o^-' a 
correct response. It v^as frequently possible to predict, on the basis'of 
first-hand experience with real students, some of the incorrect resi-onses 
that were likely to be made; thus, feedback information for the students" 
could bo made specifically relevant to these anticipated errors and the ' 
responses they would lead to. For other incorrect responses, verbal cun-, 
and hints of a more general nature were presented to the student Since 
the process of applying knowledge was regardod as primary importance 
rather than the product of such ai3plication, the student was rarely simply 
told the correct answer after a single incorrect response. For much the 
same reason, sample problems and worked-out examples for the student to 
follow were deliberately omitted from the instructional materials. ' 

in the tutorial and practice sequenges of the instructional 
program, the student was usually required to construct his own response 
even when the question itself contained strong cueing. Because of this, 
the Identification and incorporation of equivalent expressions for the 
correct answers and alno for the anticipated incorrect ones became a 
necessary part of writing the instructional materials and one to which 
the developmental evaluation made important contributions. The item=i 
for the diagnostic and mastery tests, on the other hand, were all frimed 
as multiple-choice questions. 

The instructional program was coded in Coursewriter III for use 
with the IBM -360/50 system. The bulk of the material was displayed to the 
student by means of an IBM 2740 typewriter terminal; the student also used 
this typewriter keyboard for his responses. Those portions 'of the tutorial 
sequences that seemed awkward for typing, such as, for example, the mathe-" 
matical expressions for equilibrium constants, were put on a precoded 16mm 
film strip. This teehnique was also used for the items making up the diag- 
nostic and mastery tests. An IBM 1512 random access projector displayed 
•UHuvidual frames from the filmstrip to the student as they were needed in 
the program. 
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Dmislopmenta I Kua. lua tdon 



Developmental evaluation, although it is being treated here a-s a 
separate step, procaoded concurrently with the preparation of the instruc- 
tional materials and consisted of two kinds of activities. The purpose of 
the first of thesG was to examine the objectives and test items incorporat^.d 
in the program, that of the second was to observe students using various 
parts of the instructional materials. 

Examination of objeativas and test itema. The atatements of 
ob]ectives contained in the entering skills, in the intermediate objec- ' 
tives, and in the terminal behavior were evaluated by groups of science 
and mathematics educators. These individuals assessed the appropriate- 
ness of the various instructional objectives for the courses to which they 
had been related and the suitability of the test items as measures of atu^ 
dent attainment of the objectives. 

The test items for determining the student's mastery of the 
entering skills were used with two groups of Chemistry 302 students whil.^ 
the program was being written. Based on these preliminary observations/ 
the test was revised to remove or simplify the most difficult items (thos.- 
answered correctly by less than 40% of the students) , and short reviews of 
.ome of the mathematical skills were added to the instructional progran, 

LuHvidual ohBervationa of Biudsnta. During the time that the 
instructional material?! were being prepared, a total of about PO indivi- 
duals tried out variou.5 portions of the program, ranging in length from 
short sequancea of tutorial questions up to essentially the entire program 
Those wno partiaipated at this stage as students were not always members ' 
of the defined target population, but this diversity and the critical 
acumen of some of the advanced students are conaidered advantageous in 
developmental evaluation. Graduate and undergraduate students in science 
education and in chemistry were included, and the students were observed 
closely by the author as they interacted with the instructional, materials . 

Several kinds of revision stemmed from these observations. The 
Simplest was the location and correction of errors that had^boen made in 
coding the material into the Coursewriter iii language. Additional ways 
iftf tvf acceptable answer were also identified and incorporated 

xnto the program. Similarly, new and previously unanticipated incorrect 
responses were inserted, and appropriate feedback was provided for them: 
rmally, those sequences that proved to be ineffective were completely 
rewritten, usually in an expanded form, m those cases where "conside^^ 
able revision was made in the materials, the new program was tried^out 
again with another student or group of students. 



Tea ting aM Evaluation of tJw Camp Le ted Pro^rmi 

In order to obtain quantitativo data with which to judge tho 
effectiveness of the completed prograjTi, a group of studants from ono saa- 
tion of chemistry 302 was selected to use the computer-assisted instruc- 
tional materials during the spring semes,ter, 1970. Those students were 
also expected to attend the regular class meetings of the section. Con- 
sequently, the Chi program, while it served to supplement and reinforce 
the large-group instruction in the same problem-solving and application 
skills, did not have an independent instructional role within the total 
frajTtework of the course. 

Since it was felt that a meaningful evaluation of the program 
itself depended upon the students completing it without extensive rGview- 
mg, feuch as, for exainple, in preparation for an exainination, a tLme sche- 
dule was arrived at in which the CAI program would be available for 
period of sik weeks immediately preceding an hour test that would contain 
similar kinds of problems. Because the program as developed forms a 
single cohesive unit, rather than a group of semi-independent modules, 
much of the needed data could not be obtained unless and until a student 
completed all of the progr-am. As an incentive for them to do this/ the 
cooperating professor responsible for this section granted credit for 
homework assignments zo those students who finished instructional material 

From a group of 145 students who volunteered to participate in 
the study, 35 were randomly selected. The remaining volunteers were used, 
where needed, as a comparison group. The following information was avail^ 
able for all of the students f 

(1) Level of entering skills in mathematics and in chemistry 
as measured by the entering skills test prepared as part 
of the CAI program but given, in this case, "separate from 
the program itself. 

(2) Pre-program attainment of the terminal behavior as measured 
by test items prepared for the instruction program but used, 
again ^ separate from the program itself. 

(3) Attitudes toward chemistry and toward CAI, both before and 
after the period during which the experimental group was 
using the CAI program, as measured with a semantic differ- 
ential test developed by Castleberry (1969, p, 89) for a 
similar CAI study. 



(4) 
(5) 



Hour-test scores on program^related and non^program-related 
questions . 

Final examination scores on program-related and non-program- 
related questions. 
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In addition^ fur thost? student s who completed all of the instruct 
tional program, measurements of the program path length and of the post^ 
program termin.il behavior were obtained. The program path length was defined 
as the total number of critical labels in the instructional program as used 
by a particular student, the term "critical label" denoting those labuls 
that contain a substantial ammount of cognitive content to which tho student 
must respond in some way. The post-program terminal behavior was measured 
by a set of criterion items in the last part of the instructional, program; 
thesG items were parallel with but not idisntical to the ones used to deter- 
mine the pre^program attainment of the terminal behavior. Finally, the 
members of the experimental group responded to an informal, written ques- 
tionnaire designed to elicit reactions to the total prograin. 
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RESULTS OF THE EVALUATION 



This study has involved an admittedly small and ultimately 
solf-selected student sample; consequently, it is subject to hazards 
both in interpretation and in ^generalization, and these are mentioned 
at appropriate points. These defects must not obscure the positive 
aspects. Computer-assisted instruction, especially as applied to edu- 
c,:iU.onal research, is still a relatively new field; it is also undeniably 
un expensive form of instrumentation. It can, however, be used to explore 
otherwise inaccessible queations, and Uiis exploration can lead to the 
discovery of unexpected effects. Although such effects need to be vali- 
dated by replication and by further study on a larger scale, the initial 
of tort IS not without significance. In these circumstances stringent 
-statistical tests may bo less important than the mere fact that pre- 
viously unconsidered phenomena have been given attention. 



Px'c\jram k'valuation 

The criteria for evaluating the instructional program were con- 
tainod in a set of statements of expected outcomes. it is felt that 
these outcomes, or others similar to them, are implied whenever systems 
analysis is used in the production of educational materials, in this 
case, however, they have been explicitly stated. 

(1) It ia mpeated that atudmts wUl he able to demonstrate 
aoinpatenae in tha terminal belnviop at the oonalusion of the iiistmationaL 
pTogvm, 

The mean pre-program and post-program attainment of the terminal 
behavior for the 14 students who completed the instructional program are 
shown m Table 4. The mean score on the program posttest is significantly 
lugher (f = 105, p = o.O) than the mean score on the pretest, the latter 
being adjusted to a basis of 12 items, in addition to the significant 
change m the mean score, each individual student who completed the instruc- 
tional program achieved a higher score on the post-program test than he had 
on the parallel-item test given before the progre 



ram , 
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Table 4 



MEAN ATTAINMENT OF TER4MINAL BEHAVIOR 
Mean MaKimum Possibie Range Standard Deviation 



Pre-program l.l ■ 5 0-2 

Post-program 9.4 12 S-15 



0.5 
2.0 



Table 5 summarizes the mean scoros for the experimental and the 
comparison groups on the program-related and non-program-related portions 
«f tn« third hQur-test, which was given immediately after the period dur- 

wnich the members of the experimental group were using the^CAI program 
in a. action to that for the entire experimental group, data are also included 
tor th.e^ subgroups within this group. (1) those who completed the instruc-" 
tional p.r ,ram, (2) those who started the program but did not complete it, 
and (3) chose who did not start the program. Table 6 contains th.=. corre- 
sponding uuarmation for the final examination in the course. 

^, . The mean score on the program-related items is significantly higher 

than the mean score of the comparison group in the case of the hour examina: 
tion; on the program-related part of the final examination the difference 
between the two groups is smaller and of questionable significance, when 
the.non=program-related questions on these two tests are^ considered , the 
ditterences between the two groups are small and not statistically signifi- 
cant. This comparison of performance on program-related questions using 
the total experimental group probably tends to dilute the effect of the' 
instructional program, since not all of those who were selected for the 
experimental group made equal use of the materials. 

The results for the various subgroups within the experimental 
group indicate that those students who completed the instructional program 
performed better on program-related questions and problems, both on the ' 
hour test and on the final examination, than did the members of the other 
subgroups, on the other hand, their scores on the remaining items were not 
significantly different from those made by members of the experimental ^ 
ITliTJ!^^^ if' °^ "° °- instructional program. The extent 

'incl tL h"' 5f"P^°^ students should be compared is questionable, 

since they are obviously self-selected. It was not possible, however, to 
^^r^^ f t"TT' -ith respect to mean scores on either 

standardised aptitude tests (SAT) or the preliminary tests used with this 
study (Lasater, 1971, pp. 148-149). The members of the three subgroup! 
might also be expected to differ in factors such as motivation efficiencv 
Of study habits etc. while it was not possible to measurJluchlraitr ' 

to v^^ld'^'^f -O"" expected 

to yield differences in non-program-related course perforraanoe. Such dif- 
ferences were not observed with either the hour test or the final examina- 
tion scores used in this study. examina 



Table 5 







THIRD-HOUR 


EXAMINATION 


SCORES 




Deicription of Groups 




Descriptive 
Statistici 


EKperimental 




F l^tio for Group Differences 


Total 


Subgroups^ 


Coinpari eon 
















ABC 




Total/Comp h/B B/C A/C 



Program-Related Items (Maximum ^ 36) 



Mean 


28. 2 








21.5 


9.15*** 






Range 


0-36 








0-36 








SD 


9.7 








10.5 








N 


26 








96 








Mean 




32.4 


22.8 


25.0 






6,38** <1 


1.70* 


Range 




20-^36 


0-34 


12-32 










SD 




4,9 


12.5 


11.3 










N 




14 


9 


3 











Non-Prograin-Related Items (Maximum = 64) 



Mean 


32 


4 










30.5 


<1 








Range 




S7 










2-64 










SD 


14 


4 










15.3 










N 


26 












^ 96 










Mean 






31.9 


34 


,3 


28.6 






<1 


<1 


<1 


Range 






9^60 


6^ 


59 


22-36 










N 






14 


9 




3 


i 











Subgroups I A ^ Completed the initructional program. 

B ^ Started the instructional program. 

C ^ Did not start the instructiunal prograni* 



*p > *20 (two-tailed teat) 
**p < ,02 (two-tailed test) 
***p < .002 (one-tailed test) 
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Table 6 



FINAL EXA>1INATI0N SCORES 



Descriptive 
Statistics 



DesGription of Groups 



E xpe r iine n t a 1 



Total 



Subgroups^ 



an son 



F Ratio for Group Dif ferenGes 



Total/Comp A/B B/C A/C 



Program^Related items (Maximuin ^ 36) 



Mean 


24.8 




Range 


6-36 




SD 


9.9 




N 


29 




Mean 




30. 9 


Range 




18-36 


SD 




6.4 


N 




14 




11.9*** 



Non-Program-Related Items (Maximum ^ 139) 



Mean 

Range 

SD 


93. 6 
47-132 
22.4 








91 .0 
24-139 
26.5 


<1 








N 


29 








106 










Mean 
Range 




97.7 

55- 

129 


93.2 

47- 

132 


82.4 

63- 
113 






<1 


<1 


1.66* 


N 




14 


10 


5 













Subgroups i 



A 
B 
C 



Completed the instructional program. 

Started the instructional program* 

Did not start the instructional program. 



*p > .20 (two-tailed test) 
**p < .02 (two-^ tailed test) 
•**p < *002 (one-tailed test) 



attain Uw mtmmum level of oompetenoy speaifim in elm tamUal bmiavicy^. 

Of the 3 5 students who were originally selected to use the computor- 
assisted instructional materials, 14 completed the entire program; this 
corresponds to 40% of the total starting group, if only thoso students" who 
were members of the experimental group and who were still active in the 
course at the and of the semester, i.e., who took the final examination 
are considered, the 48% of the CAI group finished the instructional program. 

The minimum criterion specified in the terminal behavior states 
that the student ■'should be able to obtain correct solutions for 70% of 
the problems representing a sampling of the above types." since the po-t- 
program criterion test contained 12 items, a score of 9 was considered 
adequate, of the 14 students who completed the program, 10 scored 9 or 
more on the criterion test; this represents 71.4%, 

(S) It is axpeated that a pr'agran havinq many branohinq iwtPts 
ba^ad on response hiatopy uiU yield a substantiaL amoiint of incHv%di,ali- 
mtion whe.n used by diver ae students, 

AA » . '^^^ ^^^^ length for students completing the CAI program was 

44 critical labels" with a standard deviation of 5 labels Figure 3 
Shows the distribution of observed path lengths. The program design itself 
sets a lower limit of 11 labels for the path length and permits, within 
the existing decision rules for branching, a theoretical maximum of about ' 
85 labels although it was not anticipated that any student would actually 
use a program of that length. The observed range of path lengths is from 
J6 to 53, the lower limit of this range being well above the minimum and 
the upper limit being considerably below the theoretical maximum; thus, 
the shortest program path length contained about two-thirds as many labels 
as the longest. ' 

is asaimed that an ability to dmonstmte the possession 
OJ- c',wta%n entertng sTclIU is a neaeBBary prerequiBite for aompletiyw the 
vnstruct^anau progrmi and mastering the terminai be?iavior. ^ ' 

The student's ability to demonstrate the possession of certain- 
prerequisite skills was measured by & written test, and the minimum 
criterion set for these entering skills was influenced more by the 
realities of what members of the target population could actually do than 
by any arguments about what thay ought to be able to do. Since the mini- 
mum criterion specified 70% correct responses, a score of 7 or more on " 
the 9=Ltem entering skills test was required as a demonstration of compe- 
tency. Twonty=two of i:he 35 students in the experimental group obtained 
such a score. Those faUing below the minimum were allowed to take the 
program anyway because an assessment of the effect of the student's ability 
to demonstrate these skills on his performance with the instructional pro= 
gram was considered an important part of the evaluation 
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The moan path length^j^ and tha moan po^Jt-procjiram tust suorvt^ for 
thofeic students who could and those who could not demons Lratu com[)Otongy 
in the entering skills aro given in thcj upp*;^r part of Tj^blo 7, it iti 
apparont that, far btiOBc otudmta uJw aatuallj aomp-.ctt.~i tlu: c>iti}\^ 
iyiDt^ruational progra/n, the mean path length for thoao who demonst . atifd 
campetency in tha entering skillB is not significantly less than the 
moan path Itsngth for those who did not demonstrato thi^^ competency. Thu 
moan posttest score was only very slightly higher for the former group 
than for the latter* 

Since observations made by the experimentGr whilQ the .ytudentts 
ware using the instructional prograin had indicated that they onQoUntered 
more difficulty with the mathematics than with the chomistry, it seemed 
worthwhile to investigate the mean path lengths and the mean post-program 
tost scoros for those students who could and could not demonstrate compe- 
toncy in the mathematics section of the entering skills test. The Srimc 
criterion of 70% correct responses was used; thus, competency waB definra 
as a score of 5 or more on the six mathematical items of the entering 
skills testa. The lower part of Table 7 presentB the results when stu- 
donts are grouped according to mathematical eompetcncy. It should be 
noted that while the groups shown in the two parts of this table contain 
the same numbers of atudents tney do not contain the same students. Both 
the difference in mean path length and the difference in mean postttast 
score have become magnified by the change in grouping, and the former 
has now become statistically significant. 

In Figure 4 the observed prograin path lengths are plotted 
against the measured level of entering skills for the 14 students who 
completed the instructional program. The equation for the least-'Squares 
straight line through the data points^ which is graphed on the same 
coordinates, is included; the quantity which is also shown, can be 
interpreted as a measure of the "goodness of fit" of the linear equation 
to the observed points. The graph of Figure 4 suggests that a relation- 
ship does exist between the student *s path length and the level of his 
entering skills; however, thi probability of obtaining this set of points 
in the absence of any dependency between the variables is large (p ^ 0.225 
with a one^tailed test) , 

Because the path length appears to be more sensitive to the mathe- 
matical portion of the entering skills test than to the total score on 
this test, the observed program path lengths were plotted against the 
scores on that part of the entering skills test that directly concerned 
mathematical objectives. Figure 5 represents this graph. The supple-^ 
mentary information with the curve corresponds to that of Figure 4. The 
higher value of ^ , as compared with that of Figure 4, indicates a closer 
agreement between the actual path lengths and the values that would be 
predicted from the linear equation. The probability of obtaining the 
observed data points with unrelated variables (p - 0,018) has become low 
enough so that the result is considered significant, particularly in view 
of the small number of cases involved. 
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TabiG 7 



MEAN VALUES OF PKOGRAM VARIABLES 



Variables 


N 




Ma an 




Range 


S 


n 


1 „ 

! /' Ratio 


— - .. — 






— 










i 




Minimurn 


Level of I 


Entering Skills 






Path Length 










[ 

t 








With 






43.5 




36-51 


4 


. 0 


<1 


without- 


3 




46,3 




37-53 


8 


.3 




Prograin Posttest 


















With 


11 




9.5 




1-12 


1 


6 


<1 


Without 


3 




9.0 




5-12 


3 


6 

■ 





Miniraum Level of Entering Skills in Mathematics 



Path Length 




! 




1 




With 


11 


' 42,5 


36-51 


■ 4,1 


8.51** 


Without 


3 


1 50.0 


48-53 


2,6 




Program Posttest 












With 


11 


9,7 


7-12 


1.8 


1,83* 


Without 


3 


8*0 


5-10 


2.6 





*p > .20 (two-tailed test) 
**p < .02 (one-tailed test) 
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SKILLS TEST, TOTAL SCORE 

FIGURE ^.--Frogram Path Ler^gth as ^ FuneEien ef the Level of 




3 4 S g 

ENTIRING SKILLS TEST, MATHEHATlCS SCORE 

figUP£ Is^-Protjrain P^th Length ai a FynetiQn ef the Level of 
Entering Skilif in Hathematiei, 
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The post-program test scores are plotted in Figure 6 against the 
modHurod level of entering skills for tiio ^tudpnts completing tho ontiro 
prograin. Here, also, the supplemGntary inform^ition is" similar to that of 
Fiqures 4 and 5 . Comparison of Figuro 6 with Figure- 4 shows "that tlmn} 
is apparently much leSB relationship between the poat-program test score:? 
and the level of entering skills than bctweGn the' path length and tlio 
level of entering skills. 

The rosuits that have been presented and discussed up to thL& 
point SGom to indicate that the student's ability to demonstrate compD- 
tency in the entering skills does not greatly influence his succgss with 
the instructional program, particularly as estimated by his score on the 
criterion test at the conclusion of the program. These results, however, 
portain only to those students who completed the materials, since it was 
only for them that the necessary data could be obtained, if all of the 
students who were selected to use the CAi program are considered, then 
it IS possible to examine whether competency in the entering skills seems 
to affect the student-s likelihood of completing tho mater'. .^.s, ThiK in^ 
formation is compiled in Table 8. Although the differences between the 
two groups of students are not statistically significant, thay do suggest 
a trend. The student who demonstrates competency in the entering skills 
soems somewhat more likely both to begin the instructional program and/ 
having begun it, to see it through to completion. At the same time, tho 
student who does not demonstrate competency in the entering skills, if 
he succeeds in completing the materials, seems to have about an equally 
good chance of mastering the terminal behavior. 




SffmtB OH Student Attitud&8 

Although the instructional program was not directly planned to 
influence student attitudes toward either the content or the instructional 
method, attitude measures were taken as a part of the evaluation. A seman- 
tic differential test was used to assess attitudes toward chemistry and 
toward the use of the computer as an instructional aid; the same test was 
administered to the entire section before and after the period during which 
the experimental group was using the instructional program. The results 
that were obtained are summarized in Tables 9 and 10, the former containing 
scores on that part of the test that was designed to measure attitudes 
toward the computer and the latter the scores for attitudes toward chemistry. 

It is seen from the data in Table 9 that the mean attitudes of 
both the CAI group and the comparison group toward the use of the computer 
as a study aid declined slightly during the experimental period, the decline 
of the CAI group being somewhat less than that of the comparison group. The 
attitudes of the two intact groups are not, however, significantly differ- 
ent for either administration of the test. The results for the three sub- 
groups within the experimental group, as summarized in the lower part of 
Table 9, show that the students who finished the program had somewhat more 
positive attitudes than did those who started it but did not finish- Inter- 
estingly enough, those students who made no use of the instructional materials 
had significantly less positive attitudes about CAI at the end of the experi- 
mental period than did those who had actually used the program. 
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EJ^TERING S. liLS TEST, TOTAL SCORE 



FIGURE 6. --Post-Program Score ai a Function of the Level of 
Entering Skills. 
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Table 8 



USE MADE 


OF CAI PROGRAM BY STUDENTS WITH 
MINIMUM LEVEL OF ENTERING SKILLS 


AND WrTHOUT 
a 






Use Made 




1 ^ 


V;ith 






Without 








N 

! 

i 


%Total %Start ^Complete 


; N 


1 

%Total 


%Start 


^Complete 


Started Instructional 
Program 




18 

! i 

' i 


82 


7 


54 






Completed Program 




i 

11 ■ 


50 61 


3 


23 


43 




Attained Criterion 






73 


2 






67 


Did Not Attain 
Criterion 




3 ; 


27 


1 






33 


Did Not Complete 
Prograjn 




7 1 


32 39 I 


4 


31 


57 




Dia Nor Start Instruct- 
ti^uiidi Prograin 


j 
j 


4 [ 


IS 


6 


46 






TO-rAL N 




22 ! 

1 




13 









^Decimal points omitted. 
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The mean scoies for the various groups and subgroups on tho na'-t 
of tho test designed to measure attitudes toward chemistry are given in 
Table 10. Again there is a slight decline in the averago" attitudes of bf.th 
the experimental group and the control group betwoen the first and serond 
testing; however, as before, there is no evidence that the two groups dif- 
fer significantly from each other. The results for the three subgroups 
within the experimental group show no significant differences among them 
although the attitudes of those who made iTiaximum use of the program ^rn 
again the most positive and those who made no use of the program show the 
least positive attitudes. 



Relationahip Between Ppogrcon VariahleB 

If all students were to demonstrate equal mastery of the terminal 
behavior at the conclusion of 'instruction, there would obviously be no 
reiationship between the student's mastery and any other instructional 
variable. Since the students who completed the present instructional pro- 
gram did not all demonstrate the same degree of mastery, it seemed inter- 
esting to ask whether attainment of the terminal behavior (as measured 
by the' post-program test) was related to the amount of instruction the 
student received (as measured by his path length through the program) . 
In Figure 7 the post-program test scores for the students who completed 
the instructional materials are plotted as the dependent variable' against 
the students' path lengths through the program,, the least-squares line 
through the points has been graphed on the same coordinates. The equation 
for this line and the value of R associated with it and the data points 
arr> also included. The probability of obtaining these results with vari- 
ables that are actually unrelated is only 0.0436 with a two-tailed tost. 
This suggests the existence of an empirical relationship between the two 
program variables. 

Since the results have already indicated that thfl path lenoth is 
probably a function of the initial ability of the student to demonstrate 
competency m the mathematical entering skills, a further reasonable ques- 
tion was whether the post-program test scorts are related to the path 
lengths for students wno have the same level of entering skills in mathe- 
matics. This was approached by assuming that the criterion test score is 
linearly related to both the path length and the level of entering skills 
in mathematics. The prediction equation that is obtained from the measured 
data under this assumption is : 



where : 
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S = 25.7 = 0,3L - 0.7M 

S ^ post-prograin test score on tenriinal behavior. 
L ^ path length. 

M ^ score on mathematical part of the entering skills test 

and where the coefficiants aave ummi compute i by the method of least 
squares. If, for a gi/en level of entering ikills, the post-prograin 
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test score and the path length arc unrelated, then the value of the coeffi- 
c'^tru^tl A fa's G^P^Cted to be zero. Tho probability of obtaining the 

calculated value under those conditions if fairly small (d = 0.0339 with a 
two-taUed test) , indicating that th. path length and tho^ post-progra^ 
^core continue to be inversely related when the level of mathematical enf 
ing skills remains constant. " " 
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Table 9 



ATTITUDES TOWARD THE COMPUTER AS A STUDY AID 



Deecriptive 
Statistics 



Deacription of Groups 


Experirnental 


Comparison 


Total 


Subgroups" 


ABC 



F Ratio for Group Differ 



ances 



Total/Comp A/B B/C A/C 



Pre-Program Scores (Maximum ^ 70)" 



Mean 


56.5 








57.7 


Range 


38-70 








39-70 


SD 


7.6 








7.4 


N 


35 








114 


Mean 




55.2 


56.1 


58.6 




Range 




38-69 


45-70 


49-70 




SD 




8.1 


7.4 


7.3 




N 




14 


11 


10 





<1 



Post-Prograin Scores (Maxiinum ^ 70) b 



Mean 
Rarge 


54.2 








51.9 


35-67 








10-70 


SD 


8,8 








12.1 


N 


27 








80 


Mean 




57.2 


54.9 


45.0 




Range 




49-67 


39-67 


35-55 




SD 




5.3 


10.0 


9a 




N 




13 


9 


5 





<1 



subgroups? A « Completad the instructional program. 

B ^ Started the instructional progrOTi* 

l^T^hi^ ' ^ ^ not start the instructional program 

This score indicates a highly positive attitude: P^^™' 

*p > »20 (two-tailed teit) 
**p < ,02 (two-tailed test) 
***p < .002 (one-tailed test) 



<1 



1.13* 



5,10** 0,76** 



EKLC 



33 



Table 10 



ATTITUDES TOWARD CHEMrSTRY 



Descriptive 
Statistics 



Description of Groups 



Experimental 


Total 


SubgroupB^ 


ABC 



Comparison 



F Ratio for Group Differe 



nces 



Total/Comp A/B B/C A/C 



Pre-^Program Scores (Maximuin = 70) ' 



Mean 


53.0 


Range 


29-70 


SD 


10.0 


N 


35 


Mean 




Range 




SD 




N 





11.4 
14 









53.5 










31-70 










8.5 


(No statistically 








114 


significant 










differences were 


52 


.9 


55.4 




found) 


37- 


63 


41-69 






9 


.6 


8.8 






11 




10 







Post-Program Scores (Maximum - 70)^ 



Mean 


49,4 




Range 


22-68 




SD 


12.4 




N 


27 




Mean 




51.6 


Range 




30-68 


SD 




11,7 


N ' 




13 




subgroups: A ^ Completed the instructional profrajn. 

B ^ Started the instructional program, 
b^, . ^ start the instructional program. 

-This score indicates a highly positive attituda. " 
*p > .20 (two-tailed tost) 
**p < .02 (two-tailed teat) 
< .002 (one-tailed taat) 
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CONCLUSIONS 



Prograni D&velopment 

The procedures that were used for developing the program and for 
designing the ingtructional materials appear to be relatively successful. 
The evidence, both from the developmental evaluation and from the formal 
testing, indicates that students do indeed acquire additional skill in 
the terminal behavior; moreover, the e^^plicit decision rules incorporated 
in the program provide variables that can be changed in specifiable ways 
for instructional research. 



Pi'ogrmi Evaluation 

The principal emphasis of the testing phase was directed toward 
assessing the effectiveness of the instructional program in terms of its 
own eKpected outcomes. Consequently, several of the cQnclusions are 
directly linked to these same expected outcomes. Some, however, are 
reiated to other aspects of program usage. 

Conclusions relating to expeatBd outQOm&B. These conclusions 
are based on observations made with a student sample that was, in effect, 
doubly self-selected. The itudents not only volunteered to participate 
in the study but also chose to complete the instructional program. The 
possible biases introduced by this are considered in the section on 
Limitations* 

■ The expected outGomes that served as evaluative criteria are 
listed in the Results of the Evaluation sectjon of this report. It was 
concluded that the program satisfied two of the expected outcomes (num^ 
bers 1 and 3) but fell short of meeting a third one (number 2) , In the 
case of the final expected outcm^ (number 4), the results proved to be 
only partially congruent. 

(1) The results of the study indicc.^e that students who complete 
the instructional program demonstrated increased competence in the temi- 
nal behavior at the conclusion of it and that this increased competence 
was maintained over at least a short period of time, 

(2) Approximately 70% of the students who completed the instruc- 
tional materials attained the minimum criterion of 70% correct responses 
as measured by th(^ progrmi posttest. 
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(3) It was concluded^ on the basis of the varianco in the path- 
length distribution, that the program provided a considerable range of 
individualization when used by a vdricty of students. 

(4) The results suggest that, at least 'undtar the conditions of 
this study, the student who is lacking in the entering skills is probably 
less likely to complete the instructional program. On the other hand, for 
those students who do finish the materials, the initial level of entering 
skill does not seem to greatly affect either the post-program test score 
or the likelihood of reaching the minimum criterion of performance. The 
student whose deficiency is in the mathamaticdl componont of the entering 
skills would, however, probably follow a longer path length and thus use 
more of the instructional material. 



CoriGlusioyis rBlating to student ahamat&ristiae arid prawm 
oariablQB. The program path length and the post--program test score are 
considered to be the important program variables, it was not possible 
to show any relationship between either of these variables and standardized 
aptitude measures (Lasater, 1971, pp. 136-138. 

(1) The student's expected path length through the instructional 
materials is inversely rGlated to his demonstrated competence in the mathe^ 
matical entering skills as measured prior to the start of the program. 

(2) There appaars to be an inverse relationship between the 
student's path length through the program (treated as the" independent 
variable) and his post-program test score (treated as the dependent vari- 
able) , and this relatidhship persists when allowance has been made for 
different levels of entering skills in mathematics. 



Canalusions relating to student atHtudeB. Althougn not a pri- 
mary part of the study, the attitude measures did yield some interesting 
results. 

(1) It appears that the instructional program might have had a 
positive effect on the attitudes toward CAl of those students who made 
maximuin use of it, 

. (2) For those who volunteered but did not make use of the pro'^ 
grain, however, a "sour grapes'^ attitude was noted. 
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DISCUSSION 



Limitations on the Study 

Several factors serve as possible limitations on the general- 
LZcibility of the conclusions baaed on the results of this study. In the 
tirat place, the students who participated wore all volunteers. Also, 
since thuy were expected to attend lactures in which the same kinds of 
applications were discussed, their motivation to oomplete the instruc- 
tional program was largely intrinsic. Thus, any extension of the results 
to a general population of students, especially when coupled with the 
extrinsic motivation that would result from a definite instructional role 
being assigned to the CAI prograin is valid only to the extent that one is 
willing to assume that the behavior of these students is typical of the 
larger group. it is the opinion in this report that those' results rsflect- 
ing primarily cognitive aspects of the student's behavior, such as the 
achicsvement of the terminal behavior, the affects of entering skills, etc 
would probably remain at least qualitatively correct. Wliare attitudinal 
or motivational effects are important, as, for example, in the fraction 
of students complating the program, there would seem to be considerably 
less certainty about any extrapolation. 

A second limitation concerns the amount of influence the instruc- 
tional program had in improving student performance with respect to .the 
terminal behavior. That such performance was better at the end of the pro- 
gram than it had been before the program was started seems undoubtfidly true- 
on the other hand, it cannot be assumed that all, or any identifiable part, 
of this learning was the direct result of the program itself, since the stu- 
dents were simultaneously exposed to instruction in the same skills during 
the class lectures. Even if it is conceded, as is warranted by the evidence, 
that the ^ students who completed the CAI program performsd at a higher level 
than their olassmatas, it cannot be assumed that this "excess" skill could 
not have been acquired by them in some other way, such as by working more 
problems from the textbook on their own. 



Impliacztione of the Study 

Inatruation and lecwning. The results indicate that students with 
less competency in the mathematical entering skills identified as specifically 
necessary for the program content followed a longer path length than did stu-" 
dents with greater competency. This is in agreement with Gagnl's generaliza= 
tion: "The course of learning for any individual is importantly affected by 
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the capabilities he brings to the instructional situation" (1965) . That the 
post-program perforinance on the criterion test was apDarently not related to 
the level of these entering skills even though the program itself provided 
no specific instruction in them suggests two possibilities. The student may 
have been relearning through practice certain capabilities that he had pre-' 
viously possessed and thus, in effect, have been improving his proficiency 
in the entering skills as he progressed through the program. It is also 
possible that the items on the criterion test, being multiple choice rather 
than of the oonstructed response form used in most of the instructional pro- 
gram, placed less demand on the student's mathematical skills; the student 
who had made a calculation e^ror might well not find his answer among the 
listed choices and in that case would be led to recheck his arithmetic It " 
should be noted again that relatively few of the students with a low level 
of entering skill completed the instructional program. 

The results also suggest that a student's criterion score is related 
inversely to his path length through the instructional program, and this depen- 
dence seems to persist even when allowance is made for differing levels of 
entering skills. This effect can be accounted for by assuming the existence 
of one or more abilities that, when present to a larger degree, permit the ' 
student to master the instructional material more rapidly and with better com- 
prehension. It is tempting to speculate that in this case the relevant ability 
might be one that facilitates the translation of verbal material into mathe- 
matical statements. If this hy|»thesis that the relationship between path 
length and post-program test score is due to the influence of differing stu- 
dent abilities of some kind is correct, then the results of the present r.cudy 
.ire m agreement with the finding of Schurdak (1965) that "more able" students 
.iGhieved higher criterion scores with lesB exposure to the course material. 

Of the 35 students originally selected to use computer-assisted ' 
instruction, only about 40% completed the entire program. This observation 
IS comparable to Castleberry • s results in a similar kind of testing experi- 
ment with first-year chemistry students; he reporta (19S9, p. 87) that" of a 
CAI sample of 100 students chosen from aiiiong a larger group of volunteers 
38 were "fully participating" and 36 had "limited participation." There is 
no real reason for assuming, however, that this same "drop-out" rate would 
persist If an instructional program had any sort of "required" status within 
a total course structure. There are a variety of possible explanations for 
the observed completion rate. Students who lacked a satisfactory level of 
entering skills may have simply found the material too difficult, the kind 
of instruction, as described by the instructional guidelines, may have been 
inappropriate for some of the students. There may have been a lack of suffi- 
cient motivation, since failure to oomplate the program was not penalized in 
any way. Finally, the students may have simply become discouraged by the 
delays and frustrations associated with malfunGtioning of the computer system 
The students themselves, when asked about this as a part of the evaluation, 
mentioned most frequently the damands on their time made by other courses or 
by other parts of the chemistry course. 
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The results obtained with the attitude inventories would seem to 
have some important i^iplications, especially for any widespread or routine 
use of CAI, Although it seenis clear that experience can affect student atti- 
tudes about CAlf it is not obvious whether the influenc© comes from the in- - 
struction itself, from the subject-matter content, from the computer used 
to manage the instruction^ or from some combination of these. The results 
seem to indicate that, at least in this study, more than one factor must 
have been important. Among those students who used some portion of the 
instructional program, the ones who completed all of ^ it had slightly more 
positive attitudes about CAI at the end of the semester than did those who 
did not complete the materials; however, the differences were not large. 
On the other hand, those members of the experimental group who did not even 
start the instructional program developed attitudes that were significantly 
less positive than the ones expressed by students who had used the materials. 
The source of these negative feelings is not at all clear. These students 
may have been influenced by the poor performance of the computer system if, 
for example, they arrived for a scheduled appointment to find that CAI was 
not operating; or they may have been affected indirectly by information about 
the program from other students, but this seems unlikely in view of the gen- 
erally more positive attitudes expressed by' the others; or their attitude 
change may have been the result of factors entirely extraneous to the CAI 
program itself, a possibility suggested by the simultaneous presence of 
less positive attitudes toward the subject matter itself. 



InatmoHoml deBign. it is the opinion in this paper that this 
program was, considering the constraints within which it was used, reasonably 
successful in meeting its instructional objectives. There remained, in spite 
of several revisions, one tutorial sequence with which students still had con- 
siderable difficulty, and this should be rewritten before further use is made 
of the program, in addition, the results of the study provide an empirical 
basis for revising and clarifying two of the eKpected outcomes. 

The preceding comments do not, of course, imply that there is any 
reason for regarding this as the "best" kind of instruction for these terminal 
objectives or for this target population. It is possible that one or more 
changes in the instructional guidelines or decision rules might improve the 
efficiency of learning, as measured either by increased student competency 
in the terminal behavior or by decreased program length, for students using 
the program. Such changes, if they are to contribute useful information for 
future instructional designers, must be explicit and must be validated by 
empirical evidence. 

Improvement in the student's ability to apply concepts and principles 
to specific situations is an important goal of most science education. The 
general structure of the program developed for this study, including the in= 
structional guidelines and the decision rules for branching, would seem to 
require only minor modifications in order to be suitable for many terminal 
behaviors involving applications of coneepts and principles ^ not only in 
chemistry but also in other areas of science. 
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Computer-assisted instruation. The opirJon expressed in this report, 
based on the experience reported, is that computar-assisted instruction pro- 
vides very valuable instrumentation for instructional research. Both the 
requirements for its effective use and the advantages it can offer have been 
adequately described in the literature (2inn, 1967; Gerard, 1967); however, 
the large invegtinents in time and money that are required will probably con- 
tinue to limit its utiliEation in this area. 

The various difficulties that were encountered with the computer 
system and their possible effects on the student users and on the results of 
this study emphasize again the importance of operational dependability. Al- 
though this has, of course, been recognized by others (Gentile, 1967; Suppes, 
1966) , the unfortunate consequences with regard to the collection and inter- 
pretation of research data are rarely mentioned. 



SuggeaHona far Fm^ther Study 

The results of this study suggest several questions that might be 
worth investigating, and these are listed below. The first four topics 
involve this particular CAI program j the other two are general areas under 
which related problems have been grouped, 

(1) What is the effect on post-program scores and on program path 
lengths of changing the decision rules to allow students to decide for them^ 
selves whether and for how long they want to solve practice problems? Does 
this change have any effect on the percentage of students completing the 
program or on student attitudes toward CAI? A study of these questions has 
been initiated, and the results will appear in a subsequent report. 

(2) What are the use characteristics, particularly in terms of 
student completion rates but also including program path lengths, post-program 
test scores, and the relationships between these program variables and the 
measured competencies of the students using the materials, of the program 
when it is integrated into the total course structure for those students who 
utilize it? How do these characteristics compare with the results of the 
present study? 



(3) What is the effect of providing remedial instruction for 
students who cannot demonstrate competency in the matheinatiGal entering 
skills? Where students do not receive remedial instruction, does their 
ability to demonstrate competency in these skills change between the begin- 
ning and the end of the instructional program? 

(4) Is the inverse relationship between program path length and 
post-program test score attributable to differences in innate ability among 
the students? Can this ability be identified and moaBured? 
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(5) Very little is known about student attitudes toward instruc- 
tional methods, yfhat factors are most influential in shaping student atti- 
tudes upon initial ex^sure to an instructional method? What is the effect 
of these initial attitudes on later experiences of the student with the same 
or a highly similar instructional method? How can negative attitudes toward 
an instructional method be altered? 

(6) There seems to be a need within instructional research for 
groups of studies in which instructional variables are ehanged one at a 
time in specific and systematic ways. These are the kinds of studies that 
should provide the empirical data needed to improve instructional theory. 
With the present study as a starting point, various directions might be 
taken. For what kinds of "app] ication" objectives can these instructional 
guidelines and decision rules be used? Are they appropriate for aecondary 
school students? Are they useful with even younger learners? If the 
instructional guidelines and decision rules are not suitable for a variety 
of objectives or for learners of different ages, can they be modified and 
made more genaral? 
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